Colorization is a computer-assisted process of adding color to black and white single images or movies. The process typically involves segmenting images into regions and tracking these regions across image sequences. Colorization is a term introduced by Wilson Markle in 1970 to describe the computer-assisted process he invented for adding color to a black and white movie or TV program. The term is now used generically to describe any technique for adding color to monochrome stills and footage.
Neither of these tasks can be performed reliably in practice, and consequently colorization requires considerable user intervention and remains a tedious, time-consuming, and expensive task. For example, in order to colorize a still image an artist typically begins by segmenting the image into regions, and then proceeds to assign a color to each region. Unfortunately, even automatic segmentation algorithms often fail to correctly identify fuzzy or complex region boundaries, such as the boundary between a subject's hair and her face. Thus, the artist is often left with the task of manually delineating complicated boundaries between regions. Colorization of movies requires, in addition, tracking regions across the frames of a shot. Existing tracking algorithms typically fail to robustly track nonrigid regions, again requiring user intervention in the process.
Thus, while the advantages and substantial commercial value of colorization in coloring black and white movies and television programs is self evident, only limited numbers of vintage movies and television programs have been colorized.
In Markle's original colorization process (Canadian Patent 1,291,260), the disclosure of which is incorporated herein by reference, a color mask is manually painted for at least one reference frame in a shot. Motion detection and tracking is then applied, allowing colors to be automatically assigned to other frames in regions where no motion occurs. Colors in the vicinity of moving edges are assigned using optical flow, which often requires manual fixing by the operator.
Although not much is publicly known about the techniques used in more contemporary colorization systems used in the industry, there are indications that these systems still rely on defining regions and tracking them between the frames of a shot.
BlackMagic, a software product for colorizing still images (NeutralTek, 2003), provides the user with useful brushes and color palettes, but the segmentation task is left entirely to the user.
Various tools are available for smoothing edges between objects on an image. In general such tools are applied globally to an entire image and not in the context of colorization or are applied to edges manually based on a perception of need by a user.
A paper entitled “Poisson Image Editing” by Perez, P., et al.; ACM Transactions on Graphics, August 2003 pages 313-318 and available at http://research.microsoft.com/vision/cambridge/papers/perez_siggraph03.pdf shows editing methods for images including a method for partial de-colorization of an image (FIG. 11).
Welsh et al. (Welsh, T., Ashikhmin, M, and Muller, K (2002) “Transferring color to greyscale images” ACM Transactions on Graphics, 21, 3 (July), 277-280, the disclosure of which is incorporated herein by reference) describe a semi-automatic technique for coloring a greyscale image by transferring color from a reference color image. They examine the luminance values in the neighborhood of each pixel in the target image and transfer the color from pixels with matching neighborhoods in the reference image. This technique works well on images where differently colored regions give rise to distinct luminance clusters, or possess distinct textures. In other cases, the user must direct the search for matching pixels by specifying swatches indicating corresponding regions in the two images. While this technique has produced some impressive results, the artistic control over the outcome is quite indirect: the artist must find reference images containing the desired colors over regions with similar textures to those that she wishes to colorize. It is also difficult to fine-tune the outcome selectively in problematic areas. Also, the technique of Welsh et al. does not explicitly enforce spatial continuity of the colors, and in some images it may assign vastly different colors to neighboring pixels that have similar intensities.
It is understood that other types of images such as line images (cartoons and the like) are easier to colorize and that in a sense they come pre-segmented.
Two published US patent applications, 2003/0194119 to Manjeshwar et al, and 2003/0113003 to Cline et al, the disclosures of which are incorporated herein by reference, describe semi-automated methods of tissue segmentation. These publications describe the use of single-voxel seeds, chosen by the user as belonging to a body tissue of interest, which are automatically expanded to find the entire connected volume of that tissue, in this case a tumor found by a PET scan, and amyloid plaque found by MRI. General Electric also sells a product called Advantage Windows, which does fully automated segmentation of bone for CT scans. In practice, it is not possible to distinguish bone voxels from blood vessel voxels with 100% accuracy; even two different doctors, attempting to do this task manually, will generally not agree on all voxels.